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Executive Summary 



School choice remains an important part of the national discussion on education reform 
strategies and their benefits. While a variety of policies encourage parents’ selection of schools for their 
children — for example, charter schools, magnet schools, and district open enrollment — scholarships that 
allow students to attend a private school have received the most attention. The U.S. Congress’ passage of 
the District of Columbia School Choice Incentive Act of 2003 in January 2004 provided a unique 
opportunity not only to implement a system of private school choice for low-income students in the 
District, but also to rigorously assess the effects of the Program on students, parents, and the existing 
school system. This report describes the first-year impacts of the Program on those who applied for and 
were given the option to move from a public school to a participating private school of their choice. 



The DC Opportunity Scholarship Program 

The 2004 statute established what is now called the DC Opportunity Scholarship Program 
(OSP) — the first Federal government initiative to provide K-12 education scholarships to families to send 
their children to private schools. The OSP has the following programmatic elements: 



• To be eligible, students entering grades K-12 must reside in the District and have a 
family income at or below 185 percent of the Federal poverty line. 

• Participating students receive scholarships of up to $7,500 to cover the costs of tuition, 
school fees, and transportation to a participating private school. 

• Scholarships are renewable for up to 5 years (as funds are appropriated), as long as 
students remain eligible for the Program. 

• In a given year, if there are more eligible applicants than available scholarships or open 
slots in private schools, scholarships are awarded by lottery. 

• In making scholarship awards, priority is given to students attending public schools 
designated as in need of improvement (SINI) under the No Child Left Behind (NCLB) 
Act and to families that lack the resources to take advantage of school choice options. 

• Private schools participating in the Program must be located in the District of 
Columbia and must agree to requirements regarding nondiscrimination in admissions, 
fiscal accountability, and cooperation with the evaluation. 
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The Washington Scholarship Fund (WSF), a 501(c)3 organization in the District of 
Columbia, was selected by the U.S. Department of Education (ED) through a competition to operate the 
Program. To date, there have been three rounds of applicants to the OSP (table ES-1). Flowever, this 
report, and the mandated evaluation of the Program, draws only on eligible applicants in spring 2004 and 
in spring 2005 (cohorts 1 and 2) and, in particular, focuses on public school applicants whose award of a 
scholarship was determined by lottery. Descriptive reports on each of the first 2 years of implementation 
and cohorts of students have been previously prepared and released (Wolf, Gutmann, Eissa, Puma, and 
Silverberg, 2005; Wolf, Gutmann, Puma, and Silverberg, 2006). 1 With the recent addition of a much 
smaller third cohort of participants, as of fall of 2006, exactly 1,800 students were using Opportunity 
Scholarships. 



Table ES-1. OSP Applicants by Program Status, Cohorts 1, 2, and 3 





Cohort 1 Cohort 2 

(Spring 2004) (Spring 2005) 


Total 

Cohort 1 and 
Cohort 2 


Cohort 3 
(Spring 2006) 


Total, All 

Cohorts 


Applicants 


2,692 


3,126 


5,818 


576 


6,394 


Eligible applicants 


1,848 


2,199 


4,047 


396 


4,443 


Scholarship awardees 


1,366 


1,088 


2,454 


396 


2,850 


Scholarship users in initial year of receipt 


1,027 


797 


1,824 


328 


2,152 


Scholarship users fall 2005 


919 


797 


1,716 


NA 


1,716 


Scholarship users fall 2006 


788 


684 


1,472 


328 


1,800 



NOTES: Because most participating private schools closed their enrollments by mid-spring, applicants generally had their 
eligibility detennined based on income and residency, and the lotteries were held prior to the administration of baseline 
tests. Therefore, baseline testing was not a condition of eligibility for most applicants. The exception was applicants 
entering the highly oversubscribed grades 6-12 in cohort 2. Those who did not participate in baseline testing were 
deemed ineligible for the lottery and were not included in the eligible applicant figure presented above, though they 
were counted in the applicant total. In other words, the cohort 2 applicants in grades 6-12 had to satisfy income, 
residency, and baseline testing requirements before they were designated eligible applicants and entered into the 
lottery. 

The initial year of scholarship receipt is fall 2004 for cohort 1, fall 2005 for cohort 2, and fall 2006 for cohort 3. 

SOURCES: The DC Opportunity Scholarship Program applications and the Program operator’s files. 



The Mandated Evaluation 

In addition to establishing the DC Opportunity Scholarship Program, Congress required an 
independent evaluation that uses “. . . the strongest possible research design for determining the 
effectiveness” of the Program. The Department of Education’s Institute of Education Sciences (IES), 
responsible for the mandated evaluation, determined that the foundation of the evaluation would be a 
randomized controlled trial (RCT) that compares outcomes of eligible public school applicants (students 



1 Both of these reports are available on the Institute of Education Sciences’ Web site at: http://www.ies.ed.gov/ncee . 
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and their parents) randomly assigned to receive or not receive a scholarship. An RCT design is widely 
viewed as the best method for identifying the independent effect of programs on subsequent outcomes 
and has been used by researchers conducting impact evaluations of privately funded scholarship programs 
in Charlotte, North Carolina; Dayton, Ohio; New York City; and Washington, DC. 2 

The RCT design for the OSP evaluation required more applications than scholarships or slots 
available in private schools, what we call “oversubscription,” to permit the random assignment of 
scholarships through lotteries. However, not all OSP applicants faced conditions for a lottery. The pool of 
eligible public school applicants in oversubscribed grades included 492 applicants in cohort 1 (spring 
2004) and 1,816 applicants in cohort 2 (spring 2005). Of those 2,308 eligible public school applicants 
who entered lotteries, 1,387 were randomly assigned to receive a scholarship (the “treatment” condition), 
and 921 were randomly assigned to not receive a scholarship (the “control” condition). The lotteries that 
generated these assignments took into account the statutory priorities, such that students from SINI 
schools had the highest probability within their grade bands of being awarded a scholarship, and students 
from other public schools had a lower probability of being awarded a scholarship. The OSP impact 
sample group includes the randomly assigned members of the treatment and control groups and comprises 
57 percent of all eligible applicants in the first 2 years of Program operation . 3 



Characteristics of Students in the Impact Sample 

Students in the impact sample were either rising kindergartners or attending DC public 
schools in the year they applied for the OSP. The characteristics of the impact sample students when they 
applied reflect the Program’s income eligibility criteria and priorities as specified in the authorizing 
legislation: 



• Their average household at the time of application had almost three children supported 
by an annual income of $17,356. 



' RCTs are commonly referred to as the “gold standard” for evaluating educational interventions; when mere chance detennines 
which eligible applicants receive access to school choice, the students who apply but are not admitted make up an ideal 
“control group” for comparison with the school choice “treatment group.” See chapter 3 for more detail on the RCT design and 
analysis. 

3 Students who were already attending a private school when they applied to the OSP are not included in the impact sample, 
although a lottery was held for those applicants in cohort L Also not included in the impact sample are the 851 students who 
applied in cohort 1 to enter grades K-5, all of whom received scholarships without a lottery because there were more private 
school slots than applicants at that grade level. 
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• Although 80 percent of their mothers reported having a high school diploma, only 6 
percent said they had a bachelor’s degree; 58 percent of the mothers reported working 
full time. 

• Nearly 90 percent were identified by their parents as African American, and 9 percent 
were identified as being of Hispanic ethnicity. 

• Twelve percent were described by their parents as having special needs. 

• They are evenly divided between males and females. 

• About 44 percent of the impact sample was attending public schools designated SIN1 
between 2003 and 2005. 

• The average impact sample student at the time of application had a reading scale score 
of 608 and a math scale score of 588, which equate to the 33 rd National Percentile 
Rank (NPR) in reading and the 3 1 st NPR in math. 

After 1 year, 77 percent of the students awarded a scholarship were attending a participating 
private school. Fifteen percent of the students who were not awarded a scholarship were nevertheless 
enrolled in a private school. As has been true in other scholarship programs, not all treatment group 
students offered scholarships choose to attend a private school, and some students in the control group 
find their way into private schools even without a Program scholarship. 

Impact sample students who used their OSP scholarship were enrolled in 47 of the 68 
participating private schools and were clustered in those schools that offered the most slots to OSP 
students. Of the students in this group, 8.4 percent were attending a school charging tuition above the 
statutory cap of $7,500 in their first year in the Program, even though 39 percent of all participating 
schools charged tuitions above the cap at that time. The average tuition charged at the schools that these 
scholarship students attended was $5,253 but varied between $3,400 and $24, 545. 4 The average OSP 
student in this group attended a school with 177 students — somewhat smaller than the average of 236 
students across the full set of participating schools. These OSP students are concentrated in the 
participating private schools with higher minority enrollments but with student/teacher ratios that are 
approximately representative of the entire set of OSP schools. Nearly two-thirds of these OSP students are 
attending participating schools operated by the Catholic Archdiocese of Washington. 

In interpreting the presence or absence of Program impacts, it is important to understand the 
difference between the treatment and control groups in their educational environments and experiences. 
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The WSF reported that families were not required to pay for tuition out-of-pocket in almost all cases where the tuition charged 
by the school exceeded the $7,500 cap. 




Examining the characteristics of the schools attended by students in the treatment and control groups 
suggests 

• There were no significant differences between treatment and control students in the 
characteristics of the public schools they attended at the time of application. 

• One year later, a similar proportion of students in the treatment and control groups 
were attending schools that offered libraries, gyms, special programs for advanced 
learners, individual tutors, art programs, and after-school programs. 

• One year later, students in the treatment group were more likely than those in the 
control group to have a computer lab or music program available to them at school. 
The treatment group was less likely to have access at school to a cafeteria, nurse’s 
office, counselors, or special programs for either non-English speakers or students with 
learning problems. 



The Impact of the Program After 1 Year 

The statute that authorized the OSP mandated that the Program be evaluated with regard to 
its impact on student test scores and safety, as well as the “success” of the Program, which we inteipret to 
include satisfaction with school choices. So far, the analysis can only estimate the effects of the Program 
on these outcomes 1 year after families and students applied to the OSP, or approximately 7 months after 
the start of students’ first school year in the Program. 



Impact of Being Awarded a Scholarship (Experimental Estimates) 

To estimate the extent to which the Program has an effect on participants, the study first 
compares the outcomes of the two experimental groups created through random assignment, called the 
“intent-to-treat” (ITT) approach. The only completely randomized and therefore strictly comparable 
groups in the study are those students whom the lottery determined were offered scholarships (the 
treatment group) and those who were not offered scholarships (the control group). The random 
assignment of students into treatment and control groups should, and did here, produce groups that are 
similar in key characteristics, both those we can observe and measure (e.g., family income, prior 
academic achievement) and those we cannot (e.g., motivation to succeed or benefit from the Program). A 
comparison of these two groups is the most robust and reliable measure of Program impacts because it 
requires the fewest assumptions to make the groups similar except for their participation in the Program. 
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The impact analysis proceeded in four steps: 



1 . The impacts of the program on each outcome of interest were estimated for the entire 
sample of study participants, using an analytic model and well-established statistical 
approaches that were specified in advance. 

2. Those same impacts were estimated for various policy-relevant subgroups of 
participants that differed based on the “need of improvement” status of their school 
(SIN1), their baseline academic performance, their gender, their schooling level, and 
their cohort status. 

3. A reliability test was administered to the results drawn from multiple comparisons of 
treatment and control group members (e.g., across 10 different subgroups) to identify 
any statistically significant findings that could be due to chance, or what statisticians 
refer to as “false discoveries. 

4. The results were subjected to sensitivity tests that involved re-estimating the impacts 
using three alternative analytic approaches. 

The findings discussed below are robust to adjustments for multiple comparisons and sensitivity tests 
unless specified. 



The analysis suggests the following findings regarding the impacts of a scholarship offer 

(table ES-2): 

• The main models indicate that the Program generated no statistically significant 
impacts, positive or negative, on student reading or math achievement for the entire 
impact sample in year 1 . One of the three alternative specifications indicated a positive 
and statistically significant math impact of 3.4 scale score points. 

• No statistically significant achievement impacts were observed for the high-priority 
subgroup of students who had attended a SINI public school under NCLB before 
applying to the Program. 

• The Program may have had an impact on math achievement for two subgroups of 
students with baseline characteristics associated with better academic preparation. The 
main models suggest that the OSP improved the math achievement of participating 
students who had not attended a SINI school by 4.7 scale score points and increased 
the math scores of those with relatively higher test score performance at baseline by 
4.3 scale score points. However, these findings should be interpreted with caution, as 
adjustments for multiple comparisons suggested they may be false discoveries. 

• No significant achievement impacts were observed for other subgroups of participating 
students, including those with lower test scores at baseline, girls, boys, elementary 
students, secondary students, or students within each of the individual cohorts that in 
combination made up the impact sample. 
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Table ES-2. Year 1 Test Score Differential ITT Regression-Based Impact Estimates 



Reading 



Student Achievement 


Treatment 
Group Mean 


Control 
Group Mean 


Difference 

(Estimated 

Impact) 


Effect Size 


/7-value 


Full sample 


606.20 


605.18 


1.03 


.03 


.56 


Subgroups: 


SINI ever 


625.50 


625.74 


-.24 


-.01 


.92 


SIN1 never 


592.14 


590.10 


2.04 


.05 


.45 


Difference 


33.62 


35.64 


-2.27 


-.06 


.54 


Lower performance 


580.89 


582.48 


-1.59 


-.05 


.65 


Higher performance 


617.19 


614.75 


2.44 


.07 


.25 


Difference 


-36.30 


-32.27 


-4.03 


-.11 


.34 


Male 


607.08 


605.51 


1.56 


.04 


.55 


Female 


605.40 


604.88 


.52 


.01 


.84 


Difference 


1.68 


.64 


1.05 


.03 


.78 


K-8 


590.80 


589.30 


1.50 


.04 


.45 


9-12 


676.23 


677.33 


-1.10 


-.04 


.73 


Difference 


-85.44 


-88.03 


2.60 


.07 


.49 


Cohort 2 


591.77 


592.15 


-.38 


-.01 


.85 


Cohort 1 


659.13 


653.03 


6.10 


.20 


.11 


Difference 


-67.36 


-60.88 


-6.48 


-.18 


.14 









Math 






Student Achievement 


Treatment 
Group Mean 


Control 
Group Mean 


Difference 

(Estimated 

Impact) 


Effect Size 


/7-value 


Full sample 


595.61 


592.87 


2.74 


.08 


.07 


Subgroups: 


SINI ever 


568.30 


568.10 


.20 


.01 


.93 


SINI never 


615.57 


610.89 


4.68* 


.12 


.04 


Difference 


-47.27 


-42.79 


-4.48 


-.13 


.17 


Lower performance 


576.07 


576.72 


-.66 


-.02 


.81 


Higher performance 


603.95 


599.66 


4.30* 


.12 


.03 


Difference 


-27.88 


-22.93 


-4.95 


-.14 


.16 


Male 


595.89 


594.61 


1.27 


.04 


.57 


Female 


595.43 


591.25 


4.18 


.12 


.06 


Difference 


.46 


3.36 


-2.90 


-.08 


.38 


K-8 


577.63 


574.86 


2.77 


.07 


.11 


9-12 


677.27 


674.67 


2.60 


.10 


.43 


Difference 


-99.64 


-99.81 


.17 


.00 


.96 


Cohort 2 


579.35 


576.16 


3.19 


.09 


.07 


Cohort 1 


655.32 


654.22 


1.10 


.04 


.74 


Difference 


-75.97 


-78.06 


2.09 


.06 


.58 



* Statistically significant at the 95 percent confidence level. 



NOTES: Means are regression-adjusted using a consistent set of baseline covariates. Impacts are displayed in terms of scale 
scores and effect sizes in terns of standard deviations. Valid N for reading = 1,649; math = 1,715. Separate reading and 
math sample weights used. 
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• The Program had a substantial positive impact on parents’ views of school safety but 
not on students’ actual school experiences with dangerous activities. Parents in the 
treatment group perceived their child’s school to be less dangerous (an impact of -0.74 
on a 10-point scale) than parents in the control group. Student reports of dangerous 
incidents in school did not differ systematically between the treatment and control 
groups. 

• The Program also had an impact on parent satisfaction with their child’s school. For 
example, an additional 19 percent of the parents of students in the treatment group 
graded their child’s school “A” or “B” compared with the parents of control group 
students. 

• For the most part, student satisfaction with their school was unaffected by the Program. 
The main exception was for students with lower test score performance at baseline, 
who on average assigned their schools significantly lower grades if they were in the 
treatment group. 



Additional Findings Regarding Using a Scholarship and Attending a Private School (Non- 
experimental) 



The results described above answer the question “what happened to OSP applicants who 
were offered a scholarship, whether or not a student used the scholarship to attend a private school?” 
Estimating the impact of using an OSP scholarship involves statistically adjusting the initial impact 
results to account for two groups of impact sample students: (1) the about 20 percent who received but 
failed to take up the scholarship offer, who presumably had zero impact from the Program, and (2) an 
estimated 4 percent in the control group who never received a scholarship offer but who, by virtue of 
having a sibling with an OSP scholarship, wound up in a participating private school (what we call 
“program-induced crossover”). These straightforward statistical adjustments yield what are typically 
called the “impact-on-the-treated” or IOT results. These adjustments increase the size of the scholarship 
offer effect estimate, but cannot make a statistically insignificant result significant. Therefore, the 
adjustments are only applied to results that were statistically significant at the scholarship offer stage of 
the analysis. 



The statistically significant findings regarding the use of a scholarship include: 



• Using a scholarship led to positive impacts on math scores for students from non-SINI 
schools (6.1 scale score points compared to 4.7 scale score points for the impact of 
scholarship award) and for students with higher test scores at baseline (5.6 scale score 
points compared to 4.3 scale score points for the impact of scholarship award). 
However, adjustments for multiple comparisons indicate both of these findings may be 
false discoveries. 
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• Scholarship use led to an average reduction of nearly a point on the 10-point danger 
perception index for parents, compared to a -0.74 point impact for the award of a 
scholarship. 

• Using a scholarship significantly increased parent satisfaction with their child’s school. 
An additional 25 percent of the parents of scholarship users graded their child’s school 
“A” or “B” compared to the parents of control group students, while the difference was 
19 percent for the impact of the offer of a scholarship. 

Estimating the effect of attending a private school, regardless of whether an OSP scholarship 
was used, also begins with the original impact results but uses a more complex statistical procedure. 6 
Because this approach deviates somewhat from the overall experimental design of the evaluation, and 
yields estimates that are less precise, the private schooling results should be interpreted and used with 
caution. Like those applied to estimate the impact of OSP scholarship use, the private schooling 
adjustments increase the size of the scholarship offer effect estimate, but cannot make an insignificant 
result significant. Therefore, the procedure is only applied to results that were statistically significant at 
the scholarship offer stage of the analysis. 



The main private schooling results suggest that 



• Private schooling was associated with higher math achievement for SINl-never 
students (by 7.8 scale score points) and for students with higher test scores at baseline 
(by 6.7 scale score points), but both of these findings may be false discoveries due to 
multiple comparisons. These private schooling differences were larger than were the 
impacts of scholarship award and scholarship use for SINl-never students (4.7 points 
and 6.1 points, respectively) and for students with higher test scores at baseline (4.3 
points and 5.6 points, respectively). 

• Private schooling is associated with lower parent perceptions of danger and higher 
parent satisfaction. The average score for private school parents represented 1.14 
fewer points or areas of concern on the 1 0-point school danger index than the average 
score for public school parents, compared to impacts of -0.74 points for scholarship 
award and a reduction of one point for scholarship use. Similarly, parents of private 
school students were 30 percent more likely to grade their child’s school an “A” or “B” 
than were parents of public school students, compared to impact on this measure of 1 9 
percent for scholarship award and 25 percent for scholarship use. 



The scholarship lottery is used as an instrumental variable (IV) to predict whether a student attended private school. Unlike an 
indicator variable for actual attendance at a private school, the prediction of private school attendance using the scholarship 
lottery instrument is unbiased because it is the same for all treatment group students (and all control group students) regardless 
of their individual enrollment decisions. 
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These results can be placed in the context of other RCTs of scholarship programs for low- 
income students, which suggest no consistent pattern of academic achievement impacts for the first year 
of program participation. Among such evaluations of four privately funded scholarship programs, one 
study of the Charlotte, North Carolina, program clearly found statistically significant overall impacts on 
math and reading for the first year, while one of three analyses of the New York City program found 
overall impacts on math achievement (Barnard, Frangakis, Hill, and Rubin, 2003; Greene 2000). When 
African-Americans are considered separately, a group that makes up nearly 90 percent of the OSP impact 
study sample, two of three analyses of the New York City program suggest there were achievement gains 
in math for African-American students in some grade levels (Mayer, Peterson, Myers, Tuttle, and Howell, 
2002), but studies of the Dayton, Ohio, and earlier District of Columbia programs found no impacts for 
this group until students were in the program for 2 years (Howell, Wolf, Campbell, and Peterson, 2002). 
In contrast, all of the randomized controlled trials that measured parent satisfaction and perceptions of 
school safety found positive impacts similar to those demonstrated by the OSP the first year (Greene, 
2000; Howell and Peterson et al., 2002). 

The findings here are based on information collected only a year after students applied to the 
Program and may not reflect the consistent impacts of the OSP over a longer period of time. Families that 
apply to voucher programs intend for their children to leave their current public schools and, in the case 
of the OSP, a much higher share of students in the treatment group (91.3 percent) switched schools — 
mostly from public to private — compared to those in the control group (56.6 percent). The first-year 
results, therefore, provide an early look at student experiences in what was a transitional year for most of 
them. Future reports will examine impacts 2 and 3 years after application to the Program, when any short- 
term effect of students’ transition to new schools may have dissipated. The later reports will also consider 
additional outcome measures, assess the extent to which school characteristics are associated with 
impacts, and examine how the DC public school system is changing in response to the Program. 
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